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ABSTRACT 



A study undertaken in Western Australia, the Western 
Australian School Effectiveness Study, explored the characteristics of 
effective high schools by investigating factors influencing science and 
mathematics achievement. A multilevel analytical model was used in the 
longitudinal study that focused on 1,024 students in 21 rural and urban high 
schools. The study confirms, as other research in Australia has suggested, 
that most variation in student achievement appears to be at the classroom and 
student level, with negligible amounts at the school level, particularly when 
prior achievement and student background effects are included in the model. 
When the classroom effects were studied, student aggregates appeared to be 
stronger than teacher effects. This study suggests that it is the classroom 
composite of peers that influences individual student achievement, rather 
than the particular teacher effects. An appendix discusses the variables and 
scales used in the analysis. (Contains 5 tables and 57 references.) (SLD) 
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Introduction 

The purpose of this study was to identify the characteristics of effective high schools in rural Western 
Austraha by investigating factors influencing science and mathematics achievement. This paper 
reports findings from two years of a longitudinal study undertaken in a cohort of urban and rural 
schools in Western Australia, called the Western Australian School Effectiveness Study (WASES). A 
multilevel analytical model was used to demonstrate that most variabiJlty in student achievement is at 
the classroom and student level, with negligible amounts at the school level. Upon further analysis of 
the residuals, this paper demonstrates that teacher effects were substantial and warrants further 
investigation. 

Background 

The concept of schools adding value to the achievement of students is used by researchers to refer to 
the extra or additional “value added” by schools to student achievement over and above the progress 
or improvement which was expected (Thomas et al., 1998; Mortimore, 1998; Sammons, Thomas & 
Mortimore, 1997). The most effective schools are those in which student progress in academic 
outcomes exceeds expectations. However, this study found little explanation for improved 
achievement at the school level. Rather, there appeared to be significant classroom and teacher 
effects, which accounted for the “value added” to student achievement. In this study, the “value 
added” to student achievement was measured while accounting for prior achievement (attainment) and 
student background. 

The “value added” model is sometimes used to analyse cross-sectional research studies, by estimating 
prior attainment using a verbal reasoning test or some other measure of student ability. This type of 
measure is flawed in that it is really measuring something other than prior knowledge. In this study, 
prior attainment was measured using a similar science and mathematics test in the previous year. This 
longitudinal approach to measuring student growth over time in science and matiiematics is now the 
new standard in educational effectiveness studies (Hill & Rowe, 1998; Sammons, Thomas & 
Mortimore, 1997, pp. 23-45). 

Research Design 

This research study, the Western Australian School Effectiveness Study [WASES] involves three 
phases (Table 1). In the First Phase, the survey instruments were developed and piloted in two 
schools in 1995 (Young, 1996; Young & Fisher, 1996a, 1996b, 1996c). 

In the Second Phase, a three-year longitudinal survey was commenced in West Austrahan high 
schools in 1996. Government, Catholic and Independent secondary schools were surveyed. Tlie 
purpose of this survey was to evaluate the school and classroom climate and characteristics of 
effective schools in differential contexts. Because the growth model is particularly useful for 
measuring change over time in student outcomes, while controlhng for other influencing variables 
which may also change over time, the same students at the same schools were surveyed over a period 
of three years (1996 to 1998). The common longitudinal cohort was a smaller sample of 1024 
students in 21 high schools. The reduction in size was due to many factors such as loss of students 
due to mobility, lack of teacher cooperation, insufficient data supplied in both years and lack of 
tracking forms by the school. Results from the WASES 1996 data collection have previously been 
reported in Young (1997a, 1997b, 1997c, 1998a, 1998b). 

Finally, in the Third Phase, a case study approach will be used to examine some exceptionally 
effective schools in the rural and urban locations of Western Austraha (1997-99). Case studies 
commenced in 1997 and selected from some outher schools based on statistical data from WASES-II 
and WASES-lil. While some interesting data has been collected and observations made at some of 
the more effective rural schools, further case study research is planned in an intensive study in 1999 
continent upon further funding from the Australian Research Council. 
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In this study, student outcomes in science and mathematics in 1997 were adjusted for student 
background, gender, academic self-concept, grade and prior attainment (1996). 

Table 1. Longitudinal sampling frame. 





* Phases 


Year 


Grade of Sample • / 


Longitudinal- ..Longituclinal 






of Study 


of Study 


■ ■ ' 


r:”- Cohort of Schools Cohort 




[WASES-I] 


1996 


Years 8, 9, 10 


28 Secondary Schools 1247 




[WASES-II] 


1997 


Years 9, 10, 11 


21 Secondary Schools 1024 




[WASES-m] 


1998 


Years 10, 11, 12 


21 Secondary Schools 1024 




[WASES-rV] 


1999 


Case Studies of Outliers 


4-8 Rural and Urban Schools/Classrooms 
- Effective and Ineffective 



The Sample 

Western Australian schools are located in a variety of locations, which have previously been 
categorized into three groups in other analyses (Tomlinson, 1994; Young, 1994a, 1994b): 
inetropolitan Perth, rural and remote. Unfortunately, these three categories did not account for rural 
cities and other types of rural locations (similarly for the remote category). Subsequently, these 
categories have been expanded by the Department of Primary Industries and Energy and the 
Australian Bureau of Statistics (DPIE 1994) into seven categories, five of which were then used in 
this study. The five categories were Metropolitan (Capital City), Small Rural Centres, Other Rural 
Areas, Remote Centres and Other Remote Areas and these were incorporated into this study. In 
Western Australia, only these five categories are applicable. 

There were 1024 students in the sample of students from 21 schools in' 1997, who were also in the 
1996 data collection and provided a complete set of information. These students were in Years 8, 9 
and 10 in 1996 and in Years 9, 10 and 11 in 1997, respectively. The initial data collection in 1996 
provided a much larger sample, however there were circumstances, which prevented a few schools 
from continuing in this study. Some of the problems included mobility of teachers and students, 
along with some schools simply withdrawing from continued participation in the study. 

Student Self-concept 

“That self-concept is related to achievement presupposes that certain classroom 
environments enhance both aspects.” (Hattie, 1992, p. 197). 

In previous research about self-concept, the multidimensional nature has been well documented 
(Byrne, 1984; Hattie, 1992; Marsh, 1990, 1993; Marsh & Shavelson, 1985). The academic 
components of the model have been the focus of attention in relationship to external constructs such as 
academic acWevement. We included two components of the Marsh Self Description Questionnaire 
(SDQH) designed to measure adolescent self-concepts (Marsh, 1992). 

^eluded in this study, was a measure of Academic Self-Concept comprised of 10 items. Examples of 
items from this measure is presented in Table 2. The Academic Self-Concept scale measures the 
student’s perceptions about their academic ability and potential to be a success at school. The 
construction of the Self-Concept scale involved the use of Confirmatory Factor Analysis and the 
method is described in a latter section of this paper. 
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Table 2. Description of some items from the Academic Self-Concept scales. 

V-' , ■ ’ Academc'Self-GoncepiScaleltems ’f 



Scale 



Academic People come to me for help in most school 10 

Self-Concept subjects. 

I’m too stupid at school to get into a good 
university. 

If I work really hard I could be one of the best 
students in my school year. 

I get bad marks in most school subjects. 

I learn things quickly in most school subjects. 



Socioeconomic Status 

Socioeconomic status of each student was constructed using two variables; mother’s occupation code 
and father’s occupation code. Students were asked to write down their mother’s and father’s 
occupation and these written responses were later coded using the Austrahan Standard Classification 
of Occupations (McLennan, 1997). These codes were later recoded using Frank Jones’ occupational 
prestige recommended codes (Jones, 1989). 

Science and Mathematics Achievement 

For the purposes of this study, a relatively simple multiple-choice test of mathematics and science was 
employed in both 1996 and 1997. This test had already been validated internationally for use in the 
Third International Mathematics and Science Study (TIMSS) for 13-14 year old students. The 
TIMSS tested and questioned students, teachers and scWls in 200 schools throughout Austraha and 
in 50 other countries. The results of the TIMSS are available from the Austrahan Council for 
Educational Research (Lokan, Ford & Greenwood, 1996) and international findings and reports may 
be viewed at the World Wide Web site: http://wwwcsteep.bc.edu/timss. 

Three different rotated forms of the possible eight tests available were used and the open-ended/free 
response part of the test was not used due to time constraints. There were 18 mathematics test items 
and 18 science test items which had to be completed in 45 minutes (total of 36 items). There was 
reading time and example test items provided prior to the commencement of the test. Analysis of the 
test items involved a procedure called Rasch Modelling which scores the test items and then estimates 
the student’s abhity on that test item as a function of the difficulty of the test item and the student 
responses to other test items. The final science and mathematics achievement measures were 
constructed using the Rasch Model. 

The Classroom Learning Environment 

That classes and schools differ in terms of their learning environments, which in turn influence 
student achievement has been demonstrated by Hattie (1987) who showed that 20% of students in 
desirable climates are better off than students in average classrooms. In the last 25 years there have 
been instmments developed for a range of classroom contexts, such as individuahsed classrooms 
(Fraser, 1990) and constructivist classrooms (Taylor, Dawson & Fraser, 1995). These instruments 
have been employed in a range of studies, with different instruments and scales used in particular 
studies. Recently, Fraser, Fisher and McRobbie (1996) developed a new learning environment 
instrument which incorporates scales that have been shown in previous studies to be significant 
predictors of outcomes (Fraser, 1994). The first version of &e new instrument contained the 
following 9 scales, each scale containing 10 items: Student Cohesiveness, Teacher Support, 
Involvement, Autonomy/Independence, Investigation, Task Orientation, Cooperation, Equity and 
Understanding. 




3 



udent Progress in Australian Schools: A Multilevel Multivariate Model 



O 

ERIC 

I 



ts 

o 

o 

C;- 

vB 

ON 

ON 



a 

o 

a 

o 

o 

r— H 

X) 

C/J 

f '■ H 

> 

f ■■ H 

O 

O 

o 

■ 

o 

cd 

(D 

T3 

§ 

4— > 

a 

<D 

T3 

c/3 



C/3 

I 

tL, 

o 



T3 

§ 

C/3 

2 



I 

cn 

o 

H 




r- 

cn 

o\ 



»n 

oo 



*o 

o\ 



cs 






1 §11 
CC {j\ 



r: 

<13 

Cu 



cs 



O 

c/3 



5 a 

o ^ 
H 2 



X3 

C3 

’S 

> 



<1> 

T3 

O 

U 



o 

o 

JC 

o 

CO 

Cm 

O 

a 

o 



CO 



CO 



o 

»n 



Tt 



o 









^ § 



o cs 

*M in 

CO CO 



VO 

VO 



cs 

Ov 



cs 

Ov 

CN 



cs 

o 



00 

ov 

c4 



a 

cd 

w 

CO 

u 



o 

c 

o 

u 

a> 

o 

‘u 

o 

CO 



oo 

VO 



CO 



CO 






a> 

> 

*tS 

<i> 

pO 

o 

U 

a 

a> 

*o 

a 

CO 






o 

ov 



q 

'd- 



ov 

Ov 



C50 



o 

ov 



Cl, 

a> 

(j 

c 

o 

U 



a> 

CO 

o 

s 




o 

o 



CO 



cs 



*n 

'd- 



ov 



»n 



VO 

o 



ov 

ov 



c 

a> 

i 

> 

a> 

S 

(j 

< 

o 

o 

G 

O 

*o 

CO 



o 

o 







* 


* 


* 


* 




* 

oo 


* 

5“ 


* 

»r> 


* 

oo 


* 


■ . 


tT 

cs 


q 

CO* 


wS 


oo 

<N 


<N 


is 


























5* 


VO 


oo 


<N 


CO 


5- 


OO 


5; 


q 


r-; 


q 


lo 


cs 


CO 


CO 







cs 

o 



oo 

»n 



ov 

CO 



o 

cs 



ov 

ov 



03 

6 

o 

> 

o 

j3 

(J 

< 

CO 



t's C 
5 w> 

J ^ 

<13 

^ 3 

Si 

D g 

of 

•C 43 

•o " 



o o 



5*3 
2 e 

C -C 

c ^ 

- sa 

o ^ 

’S'z 

.*2 5 



> > 
in 

o o 
d d 

V V 
a. a, 
eS eS 



iG CC 

'S "S 

(A *S5 



(1. U. 

U 43 

aa 

eS eS 

aa 

C/5 C/} 



JN» 



CD 



Student Progress in Australian Schools: A Multilevel Multivariate Model' * 



For the purposes of these analyses, one of the scales, Student Cohesiveness, was found to be 
sigwficantly influential in its effect on student achievement and was therefore used as an explanatory 
variable in the multilevel model. 

Comparisons of Schools by Location 

Once the scales were constructed and checked for reliability, they were compared by the five location 
categories and significant differences were found for all scales (see Table 3). The Analysis of 
Variance results showed that there were significant differences between schools from the urban and 
rural locations. 



All scales tended to be lower for students from Remote Centres, however it was suspected that these 
variations may be related to socioeconomic status. Students perceived that teachers and their own 
peers were more supportive in the country schools. Further, students from remote locations had 
significantly lower Academic Self-Concept. 

Students in country schools (rural and remote) appeared to be more satisfied with their schools. They 
felt that teachers were more supportive, friends were more supportive and generally felt safer. 
Science and Mathematics Achievement scores were not comparable due to the lack of a prior 
achievement measure in this stage of the study. That is, while there were differences in achievement 
between students from rural and urban locations, the scores were more a reflection of the students’ 
ability than a random selection. 

Methodology 

Confirmatory Factor Analysis 

These student and teacher composite scales consisted of items, which were categorical, not 
continuous. Additionally, these items varied in their loadings, which indicated that Confirmatory 
Factor Andy sis was crucial to the effective construction of the composite scale. When the observed 
variables (items) ^e non-normal and non-continuous, the use of product-moment correlations can 
lead to large negative biases in their estimates (Jbreskog, 1990; Carroll, 1961, 1963, 1989). It is 
therefore a significant feature of this study that Structural Equation Modelling techniques (WLS) were 
used, wWch assume that the observed variables are measured on an interval scale with non-normal 
distributions. Jbreskog (1994, p. 383) observed that ordinal variables represent a set of ordered 
categories, such as the five-category Likert scale, which need to be treated differently. 

The Weighted Least Squares (WLS) method available in LISREL 8 was developed to assist with the 
analysis of non-normally distributed variables by providing an appropriate weight matrix, correct 
parameter estimates, standard errors and a fit statistic. “The weight matrix required for such an 
analysis is the inverse of the estimated asymptotic covariance matrix W of the polychoric and 
polyserial correlations” (Jbreskog & Sbrbom, 1993, p. 45). 

In this study, the polychoric correlation matrix and asymptotic variance-covariance matrix were 
produced using Weighted Least Squares (WLS) PRELIS, which was then analysed using LISREL. 
This procedure was used to calculate each composite scale, assuming the one-factor congeneric 
measurement model. The one-factor congeneric measurement model (Jbreskog, 1971) was used in 
order to construct a set of factor score regression weights using LISREL (Jbreskog & Sbrbom, 
1996). Fitting a congeneric measurement model allows for differences in the contribution each 
individual measure contributes to the overall composite scale (Fleishman & Benson, 1987). 

Reliability 

That reliability is the consistency of measurement is a concept which has developed from classical test 
theory and assumes that a single true score underhes a measure (Bollen, 1989, p. 221). While 
Cronbach’s (1951) alpha coefficient is the most popular reliability coefficient in social science 
research, it has the weakness of underestimating rehability for congeneric measures. Bollen 
recommends using the Coefficient of Determination R^^j, as a viable alternative for measuring 
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reliability, where structure equations are being used. This is the measure of the proportion of 
variance in a measure, which is explained by the variables that directly effect Xj. 

For the purposes of this research, the Coefficient of Determination was used as the measure of 
reliability. The method used was based upon Werts, Rock, Linn and Jbreskog (1978). 

While Cronbach’s Alpha Reliability coefficient is provided in Table 3, the Coefficient of 
Determination is given in order to show the true reliability. Composite scales were prepared using the 
confirmatory factor analysis described above with factor score regression weights, except for 
Socioeconomic Status which was not weighted due to the different metrics used. Instead, 
socioeconomic status was calculated with unit weighting and the appropriate reliability coefficient 
used. 



The achievement test scores were constructed using Rasch modelling procedures and therefore the 
Infit Mean Square is provided as an alternative test of reliability. Further, the achievement test scores 
were kept separate by Year due to the test being different for each year, although of equal ability 
requirement. 

The Three-Level Multilevel Linear Model: Background 

While there appeared to be differences between rural, remote and metropolitan schools in the initial 
analyses, some of these differences could be due to socioeconomic factors rather than rurality. 
Further, there could be other school or teacher effects, which contribute towards explaining these 
differences. Therefore, it is not enough to simply examine location differences and report these 
individually. In order to investigate the influence of location and rurality in explaining differences in 
student achievement, a multilevel linear model of analysis was employed. In this case, a three-level 
model was used where student, class and school comprised the three levels of analysis. 

Traditional linear models on which most researchers have relied upon, require the assumption that 
errors are independent, yet most subjects are ‘nested’ within classrooms, schools, districts, states and 
countries so that responses within groups are group dependent. To ignore the nested stmcture of this 
type of data ultimately will give rise to problems of aggregation bias (within-group homogeneity) and 
imprecision (Burstein, 1980; Raudenbush, 1988). 

The Multilevel Linear Model provides an integrated strategy for handling problems such as 
aggregation bias in standard error estimates and erroneous probability values in hypothesis testing of 
school effects. For this study, MLwiN was chosen as the software program appropriate to study 
school and student effects relating to student outcomes (Goldstein et al., 1998). Research on school 
effects has previously been conducted with a set of data analysed at the individual student level, with 
the assumption that classrooms and schools affect students equally. However, when the effects vary 
among individuals and their contexts, this type of statistic^ analysis can be misleading (Bryk & 
Raudenbush, 1987). Ordinary least squares analysis provides information about the total variance, 
but can only break this total variance into the between- and within-school effects. The between- 
school effect may be influenced by school level variables, such as the affluence of the school. This 
study endeavoured to explain variations in student outcomes by first decomposing observed 
relationships into between- and within-school components. 

Previous studies have shown clearly that educational researchers need to account for the inherent 
multilevel stmcture of data collected from schools and this literature includes Mason and colleagues 
(1983), Bosker and Scheerens (1989), Bryk and Raudenbush (1986, 1989, 1992) and Goldstein 
(1984, 1987, 1995). 

The Three-Level Multilevel Multivariate Model 



The response variables for this analysis were science and mathematics achievement (each of which 
consists of 18 items). Because these response variables were conjointly affected by the explanatory 
variables, a multilevel multivariate model was selected for the analyses. A multivariate multilevel 
model is estimated by creating an extra lowest level which defines the multivariate stmcture, in this 
case science and mathematics achievement, a two-variate model. The responses were normally 
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^ more detailed discussion of this methodology is found in Goldstein and colleagues 
(1998, pp. 77-85) and Goldstein (1995, pp. 69-76). 

In the model estimated here, each student has two records: either mathematics achievement or science 
acluevement. These are the within-student measurements and there is a pair of dummy variables 
which indicate which response variable is present (z,: l=maths, 0=science; z{. l=science, 0=maths). 
A sunphfied model is written below with an explanatory variable, xj^i. The y^y^t/ represents the 
response variable for maths or science (level- 1: /), for student j, in classroom k and school 1. 

yijkl = Poikl^lijkl + Po2kl^2ijkl + Pllkl^lijklXjkl + Pl2klZ2ijkl^jkl + Vy/ + V2/ 

+ + U2kl+ ^Ijkl + ^2jkl 

var(vyy) = 0^,2 var(v 2 /) = 0 , 2 ' cov(vyy V 2 /) = a ^,2 

yar(uiki) = oj var(w2^/) = cj cov(wyy.y 1*2^./) = a„,2 

= y^(e2jki) = cj C0V(eyyy,ye2yw) = O.,2 

Equation 1 



There was no level- 1 variation in this model, because the model existed solely to define the 
multivariate stmcture and level- 1 defines the dej^ndent variable: science achievement or mathematics 
achievement. The level-2 variances and covariances were the residual between-student variances: 
student level. The level-3 variances were residual between-class variances: teacher or classroom level; 
while the level-4 were between-school variances: school level. There were no missing data in this 
cohort. 

Results of the Multilevel Multivariate Model 
Model 1: Null Model 

The multivariate model was used to simultaneously analyse mathematics and science achievement. 
The three-level model (student, class and school) was analysed firstly as a null model. This model is 
actually a four-level model, with the first level being a dummy variable indicating maths or science 
achievement. There were no explanatory variables in this model. The purpose of this analysis was to 
decompose the total unexplained variance in science or mathematics achievement into the three 
variance components. These results are presented below in Table 4, where it is clear that there is litde 
variabUity between schools. That is, most of the variation in achievement occurs at the student and 
classroom levels. For science achievement, there was 67.3% variation between students, 27.8% 
between classes and 4.9% between schools. Similarly, there was 64.1% variation in mathematics 
achievement between students, 27.9% between classes and 8.0% across schools. The likelihood ratio 
statistic {-2*Log(likelihood)} was 4739.027 for science and mathematics achievement as a 
multivariate dependent model. So this analysis demonstrates the degree to which classroom effects 
appear to outweigh school effects. 
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Table 4. Parameter Estimates and Standard Errors for the Multivariate Multilevel Analysis of 
Science and Mathematics Achievement (Students, Classes and Schools): 

Model 1 (Null Model) and Model 2 (Gender, SES, Grade, Location, Prior Achievement 
and Academic Self-Concept). 





Model P 


Model 2’ 




Science 


; \Matheraatics 


Science 


‘ ; Mathematics . . ‘ii 




Achievement 


"Achievement 


Achievement 


Achievement n 


Fixed Parameters 


Estimate (s.e.) 


Estimate (s.e.) - 


Estimate (s.e.) 


Estimate (s.e.). ; 


Intercept 


1.108 (0.110) 


1.294 (0.126) 


-4.718 (1.409) 


-4.272 (1.099) 


Gender 






.094 (.073) 


.008 (.066) 


Socioeconomic Status 






.023 (.031) 


.048 (.028) 


Grade (9,10,11) 






.496 (.145) 


.434 (.112) 


Location 






-.075 (.061) 


-.111 (.049) 


Achievement 1996 (Sci/Maths) 






.180 (.041) 


.337 (.028) 


Academic Self-Concept 






.272 (.047) 


.336 (.043) 


Random Parameters 


Variance Estimate 


•Variance Estimate 


Variance Estimate 


Variance Estimate ^ | 


School Variance 


0.077 (0.080) 


0.135 (0.104) 


0.041 (0.049) 


0.030 (0.031) 


School Covariance 


0.108 (0.086) 




0.036 (0.034) 




Class Variance 


0.437 (0.110) 


0.471 (0.118) 


0.248 (0.070) 


0.127 (0.042) 


Class Covariance 


0.386 (0.103) 




0.117 (0.044) 




Student Variance 


1.056 (0.054) 


1.080 (0.056) 


0.972 (0.050) 


0.811 (0.042) 


Student Covariance 


0.522 (0.043) 




0.325 (0.034) 




-2 log(likelihood) 


4739.027 


4497.915 


Deviance* 








241 




Variance at Each Level for the 


Variance Explained by Model 2 





Null Model 1 (%') 


Pnrjimpfprc 


School Level 


4.9 % 


8.0 % 






Class Level 


27.8 % 


27.9 % 






Student Level 


67.3 % 


64.1 % 






Total Variance Explained 


100.0 % 


100.0 % 


19.7 % 


42.6 % 



The deviance statistic is the difference in -2Iog(Iikelihood), which is a chi-squared distribution and significant. 
^ N = 1024 students within 21 schools 



ERIC 

MMiffilffTlTLiU 



BEST COPY AVAlUBLi 



8 



Student Progress in Australian Schools: A Multilevel MultivWiaiTModel — " ' 



Model 2: Context and Student Background Model 

The next multivariate analysis used a four-level model, the first level is not referred to here in these 
discussions as it indicates which achievement variable is being estimated (science or maths). The four 
levels included the response variable (science or maths), the student level, the class level and the 
school level. 



Mathematics or Science Achievement 
Student Level 
Classroom Level 
School Level 



In preliminary analyses, the explanatory variables included gender, socioeconomic status, grade, 
location, previous achievement and student academic self-concept. These were largely student 
background variables, with grade to compensate for the three grade levels analysed, location to 
measure the effect of the location of the school (rurahty) and previous achievement being the science 
and mathematics achievement score from the 1996 WASES data set. 

Results from these analyses, provided in Table 4 Model 2, demonstrated that these variables 
accounted for 19.7% and 42.6% of the total unexplained or residual variation in science and 
mathematics achievement, respectively. However, the significance of Gender, SES and Location was 
minimal. These three variables were removed from the next two estimated models. The likelihood 
ratio statistic was now 4497.915 for science and mathematics achievement. The difference was a 
deviance statistic, which was statistically significant at 241 (a chi-squared distribution on six degrees 
of freedom - that is, six additional fixed parameters were estimated). School location was found to 
have httle effect upon student achievement once student background and prior achievement was 
accounted for. This is an important finding in this study, as there were demonstrated significant 
differences in achievement across different locations in previous analyses. However, it appeared that 
the significant effects of academic self-concept, grade and prior achievement scores on a similar test 
accounted for most of these differences. 

Models 3 and 4: Classroom and Peer Model 

Further to the previously estimated Model 2, a reanalysis was conducted with three explanatory 
variables, which had been demonstrated to be statistically significant in explaining variations in 
student achievement. Model 3, therefore, included Academic Self-Concept, Previous Achievement 
and Grade. 

Results from Model 3 provided a baseline for comparing the effect of three classroom variables in 
Model 4, that is. Cohesion, Socioeconomic Status and Academic Self-Concept. These three variables 
were classroom average measures of the degree to which students felt their classroom was cohesive, 
the average socioeconomic status of the students in the classroom and the average academic self- 
concept of the students. 
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Table 5. Parameter Estimates and Standard Errors for the Multivariate Multilevel Analysis of 
Science and Mathematics Achievement (Students, Classes and Schools): 

Model 3 (Academic Self-Concept, Prior Achievement and Grade) and Model 4 
(Cohesion, Average SES and Average Academic Self-Concept). 

Model 3' Model 4* 





Science 

Achievement 


Mathematics 

Achievement 


Science 

Achievement 


. - Mathematics •. 

^ ' Achievement' : 


Fixed Paramcicrs 


. Estimate (s.e.) 


Estimate (s.e.) 


Estimate (s.e.) ' 


Estimate (s"e.).' ' \ 


Intercept 


-4.801 (1.401) 


-4.539 (1.083) 


-7.975 (1.382) 


-7.133(0.991) 


Academic Self-Concept 


0.273 (0.047) 


0.342 (0.043) 


0.243 (0.048) 


0.305 (0.045) 


Achievement 1996 (Sci/Maths) 


0.191 (0,040) 


0.345 (0.028) 


0.191 (0.040) 


0.345 (0.028) 


Grade (9,10,11) 


0.492 (0.146) 


0.439 (0.112) 


0.459 (0.132) 


0.380 (0.091) 


Classroom Cohesion 






0.320(0.189) 


0.405 (0.151) 


Classroom Socioeconomic 
Status 






0.281 (0.126) 


0.045 (0.100) 


Classroom Academic 
Self-Concept 






0.446 (0.187) 


0.455 (0.147) 


Random Parameters 

i 


Variance Estimate 


.Variance Estimate 


Viiriance Estimate . 


X 

■Variance Estimate .. ;| 


School Variance 


0.043 (0.050) 


0.053 (0.039) 


0.000 (0.000) 


» 1 * r:. J 

0.030 (0.020) 


School Covariance 


0.045 (0.038) 




0.000 (0.000) 




Class Variance 


0.257 (0.072) 


0.126 (0.042) 


0.171 (0.046) 


0.047 (0.024) 


Class Covariance 


0.123 (0.045) 




0.053 (0.025) 




Student Variance 


0.973 (0.050) 


0.813 (0.042) 


0.981 (0.050) 


0.823 (0.042) 


Student Covariance 


0.322 (0.034) 




0.331 (0.035) 




-2 log(likelihood) 


4506.953 


4477.895 


Deviance “ 








29 



Variance Expj^ned by Model 3 , Variance Explained by Additional ' 
Parameters (%) Model'4 P^arneters^(%) 

Total Variance Explained 18.9% 41.2% 7.7% 5.5% 



The deviance statistic is the difference in -21og(likelihood), which is a chi-squared distribution and significant. 
N = 1024 students within 21 schools 
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Discussion 

The analyses presented in this paper were based on data collected from students and teachers 
participating in the Western Australian School Effectiveness Study in 1996 and 1997. The cohort of 
students was longitudinal, yet smaller than had been hoped. While providing a rich source of 
information, the size of the sample became reduced as missing data was removed and students 
dropped out of the study. Yet, this was a valid cohort, which provided useful information about what 
makes a school effective. 

Student Level. This study demonstrated that, while rural differences were apparent in student 
outcomes such as science and mathematics achievement and academic self-concept, these differences 
were of no consequence when investigated using sophisticated multilevel modelhng techniques. That 
is, rural schools were not disadvantaged by their location. Rather, rural students were disadvantaged 
by their self-concept. Students in rural schools did tend to have a weaker belief in their own academic 
ability to perform, irrespective of their actual ability. Relative to prior achievement, self-concept and 
grade, this study demonstrated negligible gender differences and socioeconomic differences in 
achievement. 



° Brevipus Achievement 

• Perception of Academic Ability 

• Grade 



Class Level. There were sizeable variations in science and mathematics achievement across classes, 
with little found between schools. When these classroom differences in achievement were examined 
further, individual student effects accounted for a big portion of the class-level differences (40% for 
science; 70% for maths) with other students in the classroom accounting for a lot as well (20% for 
science; 17% for maths). Classroom effects that were significant in explaining differences in 
achievement included: 



e Achievement of Peers 

• Previous Achievement of Peers 

• Perception of Academic Ability of Peers 

• Classroom Cohesiveness 

• Grade 



School Level. This study demonstrated negligible school level differences in achievement, with 5% 
in science achievement and 8% in maths achievement varying across schools. These small differences 
were entirely accounted for by the student and classroom level variables described previously. 
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Limitations of the Study 

Student achievement in mathematics and science was measured as an essential and unique feature of 
this study. While difficult, time-consuming and expensive, this approach provided the study with the 
same achievement measure for all students. Teachers who participated in this study were active 
partners in collecting and coordinating the testing procedure. However, it should be noted that this 
study required more funding than other comparable studies of school effectiveness. Additionally, in 
Western Australia the rural high schools are spread much further apart making it much more difficult 
to visit them without expending considerable fonds in travel costs. It would have been advantageous 
if more funds had been available for supporting this study; however the competition for grant funding 
is critical. 

This study was limited not only by financial and strategic considerations in the management of a large 
study, but also by problems with student mobility. Further complicating effects include the primary 
school attended and neighbourhood membership. Hill and Goldstein have developed an approach for 
modelling “cross-classified” (Hill & Goldstein, 1998; Goldstein, 1995, pp. 113-124) students. This 
is obviously an important step to improve the “school effectiveness statistical models”. That is, 
variations in student outcomes may be attributed to the neighbourhood where the child comes from 
and/or the primary school previously attended, as well as the high school currently attended. A single 
school may contain children from many neighbourhoods and primary schools. Further, a single 
student may attend multiple schools throughout the course of the study. In the WASES study, these 
students simply dropped out, with subsequent data loss. 

These limitations have highlighted the need for an improved research design in future studies. 
Improvements in financial planning, strategies for retaining the cohort in a longitudinal study and 
attention to quality control aU point to a new longitudinal study of school effectiveness and 
improvement in Western Australia. 

Conclusions 

Student achievement varies between students, classrooms and schools. These variations have been 
investigated using multilevel modelling techniques, however they are not fully explainable by these 
procedures. That school effects on student achievement tend to be minim al in Australia, has already 
been found by Hill and Rowe (1996, 1998) in another Australian study, the Victorian Quality Schools 
Project (VQSP), as well as earlier studies by Young (Young, 1998b; Young & Fraser, 1993). These 
studies confirm that most variation in student achievement appears to be at tiie classroom and student 
level, with negligible amounts at the school level particularly when prior achievement and student 
background effects were included in the model. When the classroom effects were investigated, 
student aggregates appeared to be stronger than teacher effects. That is, this study suggests that it is 
the classroom composite of peers which influence individual student achievement, rather than the 
particular teacher effects. 

Further analysis of teacher and classroom effects in a longitudinal, school effectiveness and school 
improvement model are strongly recommended in order to understand the effect of teacher and 
classroom variables on a variety of student outcomes. 
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APPENDIX A 



Variables and Scales used in the Multilevel Multivariate Model 



Variable 


Coding and Recoding 


Science 

Achievement 


A student science achievement test consisting 
of 18 multiple choice response format test 
items selected from the Third International 
Mathematics and Science Study (TIMSS) and 
total score estimated using the Rasch Model. 




There were two measures used here: science 
score for 1996 and 1997. 


Mathematics 

Achievement 


A student mathematics achievement test 
consisting of 18 multiple choice response 
format test items selected from the Third 
International Mathematics and Science Study 
(TIMSS) and total score estimated using the 
Rasch Model. 




There were two measures used here: maths 
score for 1996 and 1997. 


Ach96 


Science and mathematics achievement scores 
from 1996 estimated separately using the 
Rasch Model (see above). 


SES 


Socioeconomic Status of the students 
consisting of mother and father’ s occupations 
and education (continuous and standardised). 


Gender 


1 = males; 0 = females 


Self-Concept 


A measure of the students’ self-concept: 
Academic Self-Concept (continuous and 
standardised). 


Class Cohesion 


A measure of the students’ perceptions of the 
classroom learning envirorunent kept at the 
student level (continuous and standardized). 


Location 


A five-category measure described previously: 
Metropolitan Perth, Small Rural Centre, Other 
Rural Areas, Remote Centre and Other 
Remote Areas ( 1 to 5). 
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